ISIT 2015 Tutorial: Information Theory and Machine Learning

We are in the midst of a data deluge, with an explosion in the volume and richness of data sets in fields including social networks, biology, natural language processing, and computer vision, among others. In all of these areas, machine learning has been extraordinarily successful in providing tools and practical algorithms for extracting information from massive data sets (e.g., genetics, multi-spectral imaging, Google and FaceBook). Despite this tremendous practical success, relatively less attention has been paid to fundamental limits and tradeoffs, and information theory has a crucial role to play in this context. The goal of this tutorial is to demonstrate how information-theoretic techniques and concepts can be brought to bear on machine learning problems in unorthodox and fruitful ways. We discuss how any learning problem can be formalized in a Shannon-theoretic sense, albeit one that involves non-traditional notions of codewords and channels. This perspective allows information-theoretic tools—including information measures, Fano’s inequality, random coding arguments, and so on—to be brought to bear on learning problems. We illustrate this broad perspective with discussions of several learning problems, including sparse approximation, dimensionality reduction, graph recovery, clustering, and community detection. We emphasise recent results establishing the fundamental limits of graphical model learning and community detection. We also discuss the distinction between the learning-theoretic capacity when arbitrary “decoding” algorithms are allowed, and notions of computationally-constrained capacity. Finally, a number of open problems and conjectures at the interface of information theory and machine learning will be discussed. ∗Program in Applied and Computational Mathematics, and Department of Electrical Engineering, Princeton University, Princeton, USA, [email protected], www.princeton.edu/∼eabbe †Departments of Electrical Engineering and Computer Science, and Department of Statistics, University of California at Berkeley, Berkeley, USA, [email protected], http://www.cs.berkeley.edu/∼wainwrig.